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Diagnosing Knowledge States in Algebra 
Using the Rule Space Model 

Abstract 

This paper illustrates the use of rule space as a tool to support cognitive analyses of students* 
mathematical behavior* The rule space approach is explained and is then used to classify students 
into one of two methods for solving linear algebraic equations in one imknown and to diagnose 
their knowledge states in this topic* A 32-item test with open-ended questions was administered to 
231 eighth and ninth graders- The following outcomes of the rule space model were presented: (a) 
a classification of examinees into knowledge states resulting ftom the two solution approaches at 
the group level along with individual examples; (b) tree-diagrams of the transitional relationships 
among the states for each strategy* Implications for using the feedback provided by the rule space 
model in the context of instruction and assessment aie discussed* 
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Diagnosing Knowledge States in Algebra 
Using the Rule Space Model 

Indices that may be quickly and inexpensively generated from standard mathematics tests 
include total number correct for each student^ measures of central tendency, dispersion, and 
standard errors of measurement While these summary and descriptive statistics are of value in 
ranking students or comparing a student's performance to the performance of students on some 
larger ncsrmative sample, they do not provide much diagnostic information about the mathematical 
operations that the student has mastered or has not yet mastei^ 

In the case of soluticm of linear equations in one unknown, for example, a teacher may 
wish to know more than that a given student is "poor at algebra" because his or her score was one 
standard deviation below the class mean. Ideally, the teacher would wish to know which of the 
many components of performance in algebra is causing difficulty for a given student and for the 
class as a whole. Adequate performance in the algebra of linear equations requires more than skill 
in applying an algorithm. It rests upon adequate performance in and understanding of a larger 
body of mathematics that ranges from mastery of simple operations such as addition to mastery of 
more difficult concepts such as the distributive law and quotients. Armed with this diagnostic 
information, the teacher may then examine the difficult area(s) for the students in terms of 
misconceptions or faulty skill performance using any of the interview and protocol analytic tools 
provided by researchers in cognitive science. In considering group-level performance, the teacher 
may wish to examine teaching methods to determine if these arc responsible in any way for the 
students' misleaming. 

The value of a diagnostic profile that points out deficiencies and strengths in the students' 
performance in mathematics has long been recognized (VanLehn, 1982). However, the 
computational problems involved in "teasing out" the dimensions underlying students' 
performance arc formidable. VanLehn (1982) noted that thousands of hours of work by trained 
experimenters were required to determine students' "bugs" in subtraction. The problem is 
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exacerbated when the instructor has available only the student's COTiectAnconect score on each 
item, or has little time to deal with detailed levels of assessment 

To illustrate the combinatoric problem involved in producing profiles of mastery/non 
mastery on task subcomponents, imagine that one can describe the solution of an item in terms of 
the mastery of four underlying dimensions. Thus, it may be argued that a student who fails this 
item (who is not gu::.ising) may have failed to master all four dimensions, or failed to master any 
three, or any two, or any one. For a problem with four dimensions, there are 2^ -1 patterns that 
could account for an incorrect answer. In general, for an item with k dimensions, there arc 2^^ -1 
patterns that could account foe an incorrect answer* As the number of dimensions increases, the 
number of patterns to consider climbs exponentially* 

Tatsuoka developed the rule space methodology to address the combinatoric problem 
associw )d with diagnosis of mastery of underlying dimensions of an item (called attributes), (e*g., 
Tatsuoka, 1983, 1985, 1990, 1991; Tatsuoka & Tatsuoka, 1987)* An attribute of a task is a 
description of the processes, skills or knowledge a student would be required to possess in order 
to successfully complete the target task. Attributes are not generated by rule space; they arc 
generated by a domain expert (usually in concert with a cognitive scientist). They may include, 
but are not limited to, a student's ability to perform some procedures. Attributes may also include 
a student's use of heuristics, or adoption of a strategy* In general, rule space can handle any 
expressicMi of an underlying dimension of a task that can be specified to the extent that certain items 
tap that attribute and other that do not. By examining a student's differential performances on the 
items, rule space categorizes students into the attribute mastery pattern that would best account for 
the student's individual, item-response pattern. 

For the mathematics educator, rule space can provide the following diagnostic information: 
(a) a description of each student's mastery (and nonmastery) of the attributes judged by a domain 
expert to be necessary for successfully completing the test; (b) a description of group level mastery 
patterns obtained by aggregating across the individual profiles; and (c) partial-mastery charts that 
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can be used to aid in the design of remediation (Tatsuoka & Tatsuoka, 1992). A detailed 
description of rule space is beyond the scope of this paper. A simplified description follows. 

Rule space is a statistical methodology for classifying students' responses to a set of items 
into one (or nioie) piespecified attribute-mastery patterns. In practice, a domain expert and 
cognitive scientist would identifj' the attributes of the target task that are of interest They would 
then write items that sample from this set of attributes. The resulting items and attributes would 
then be arranged in an attribute-by-item matrix (referred to as a O matrix in rule space). 
Unfortunately, a student's actual mastery or nonmastery of a set of attributes cannot be measured 
directiy, but must be inferred from the student's pattern of responses to the items. In an ideal case, 
a student who had mastered some, but not other, attributes would answer correcdy only those 
items that contain attributes that he or she had mastered, and answer inconecdy those items that 
contain at least one attribute that he or she had not mastered. Such a student would produce an 
ideal item-response pattern . Within rule space, specialized functions, called Boolean Description 
Functions (BDF), are used systematically to determine the knowledge states of interest (i.e., those 
that describe ideal behavior in terms of attributes) and to map them into ideal item-response patterns 
(Tatsuoka, 1991; Varadi & Tatsuoka, 1989). Rule space then plots the ideal item-response 
patterns in terms of two variables: 0 (theta), and ^ (zeta). 

6 and The ability continuum derived from an item-response (IRT) analysis (Lord & 
Novick, 1968), 6, is used as one dimension along which to describe the ideal item-response 
patterns. Thus, a high-ability student (scoring high on 9) would have an ideal item-response 
pattern with many Is and few Os (for correct and incorrect responses to items, respectively); 
conversely, a student at the lower end of the ability continuum (scoring low on 0) would display 
an ideal item-response pattern containing mosdy Os. A student of high ability who gets some easy 
items incQtiect, or a student of low ability who gets some hard items correct would be measured 
high on an "unusualness of response" scale, which is what ^ is (Tatsuoka, 1984; Tatsuoka & 
Linn, 1983). ^ is the second dimension that rule space uses to describe students' responses. 
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Thus, rule space generates a two-dimensional cooidinate space (with G on the x-axis and ^ 
on the y-axis) in whose plane certain points represent the 6 and ^ of the ideal-response patterns. 
However, students' performances on ihe test items are often subject to fluctuations. Producing an 
ideal response pattern is likely to be rare. Students' item-iesponse patterns that deviate from an 
ideal response pattern are considered as "fuzzy" response patterns. Points corresponding to the 
fuzzy response patterns swarm around their respective ideal response patterns, and generate 
regions within probability ellipses with the ideal response patterns as their centers. A 90% 
probability ellipse encloses 90% of the fuzzy-response-pattem points; a 95% probability ellipse 
encloses 95% of them; and so forth. Rule space then uses Information on a student's actual score, 
measured on 6, and ^, to decide where in the two-dimensional space spanned by these measures 
the student's fuzzy item-response pattern lies (Tatsuoka & Tatsuoka, 1987). A student is 
classified to the ideal response pattern that embraces his or her point in the smallest associated 
ellipse. This determination is made by measuring how far from the centroid the student's point is, 
in terms of Mahalanobis' distance. Once the most likely ideal item-response pattern is identified, 
the most conservative attribute-mastery pattern for tliat ideal item-response pattern is assigned by 
rule space to that student. The most conservative pattern is chosen for instructional purposes. The 
most conservative pattern will err in the direction of suggesting that a student has not mastered the 
identified attributes, when he or she may have mastered them. Thus, the conservative diagnosis 
would spur a remedial strategy that would be most likely to target the student's weaknesses. 

Rule space entails a statistical pattern classification approach. Its accuracy of classification 
depends on how well the items arc written, how well they test (as unambiguously as possible) the 
attributes that were established by the domain expert, and the amount of error in the student's 
responses. Since rule space does not produce the attributes, the onus lies on domain experts and 
cognitive scientists to provide it with useful descriptions. For areas that are well-defined (e.g., 
subtraction of fractions, signed numbers operations), rule space has been shown to perform quite 
well (Tatsuoka, 1990; Tatsuoka & Tatsuoka, 1992). 
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From among the several methods possible to solve a giv^^n linear equation in one unknown, 
we have chosen to demonstrate the use of rule space using two different approaches (expressed as 
two different Q matrices). One method involves the use of a simple heuristic - initially evaluating 
the equation to determine if a simpler solution path would result by not rewriting the equation in 
standard form until the final step (method I). The other method involves consistently rewriting the 
equation in standard form (i.e., with variables on the left-hand side of the equation and constants 
on the right (method II). More details regarding the two methods are given in the method section. 
The purpose of the present study was to illustrate the application of the rule space model for 
diagnosing students* knowledge states in linear equations based on the two specified solution 
methods. Thus, we will sec how rule space can'be used to identify students who may need further 
remediation, to identify subcomponents of linear algebra that may be causing difficulty for the 
entire group of students, to produce partial mastery charts that may form the basis of fruitful 
remediation, and to identify students for whom it may be of value to study further in terms of their 
particular strategy use. 

Method 

Subjects 

The sample consisted of 231 8th and 9th graders (age 14-15) from an integrative high 
school in Tel Aviv. Fifty-seven percent of the subjects were girls. The students studied 
mathematics in high and low achievement groupings (106 in the former and 125 in the latter). 
Instruments and procedures 

A 32-item diagnostic test in linear algebraic equations in one unknown was developed by 
Gutvirtz (1989). (The test items appear in Appendix A). 

The internal consistency of the 32-item test as measured by Cronbach's Alpha coefficient 
was 0.95. The item difficulty indices (percent correct) ranged from 0.41 to 0.93 with an average 
of 0.74. The item discrimination indices (item-total correlations) ranged from 0.40 to 0.75 with an 
average of 0.60. 
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Two sets of attributes were specified for the two solution methods (see Tables 1 and 2) and 
these sets used to produce two separate Q matr ces (see Appendices A & B). The two sets of 
attributes lesult from a strategic decision made at the outset In method I, a heuristic/'evaluation," 
is applied, wherein the student scans the equation in its initial form to determine if it is likely to be 
simpler to delay writing the equation in standard form until the final step. For example, the 
evaluation rule could be applied to item 5 in the test When the evaluation heuristic is applied the 
solution unfolds, thus: 

4x + 21= lOx + 17 

21 - 17 = lOx - 4x (evaluating, and subtracting an x-term and a constant from both sides) 
4 = 6x (adding or subtracting variable terms) 

6x = 4 (applying the symmetry law) 

X = 4/6 (dividing across by the coefficient of x, when a>h) 

These operations are denoted in Table 1 as 12, 2, 6, 11, 10, and 14 (see also the 
corresponding row for item 5 in Appendix A). 

In method H, the student performs the mathematical operations necessary to bring the x- 
tcrms to the left-hand side of the equation, and the constants to the right in all cases. Thus when 
the evaluation heuristic is not applied, the solution path is more complex, since it now involves 
operations with signed numbers: 
4x + 21 = 10x + i7 

4x - 10 X = 17 - 21 (subtracting an x-term and a constant from both sides) 

-6 x = -4 (adding or subtracting variable terms and operations with signed 

numbers) 

6x = 4 (multiplying both sides of the equation by -1) 

X = 4/6. (dividing across by the coefficient of x) 

Tncse operations arc denoted in Table 2 as 2, 7, 8, 11, 13 (see also the corresponding row 
for item 5 in Appendix B). 
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The rule^spacft analysis: 

1. The adequacy of the two attribute matrices was tested by regressing the vector of item 
difficulties on the set of attribute vectors. The entire set of attributes accounted for 95% of the 
variance (R2=.95; R^adj--^!) for method I, and 77% of the variance (r2=.77; R2adj=-63) for 
method n in the total sample. 

2. The BILOG program (Mislevy & Bock, 1983) was used for estimating the item 
parameters (a's and h's) of the IRT two-parameter logistic model. The a values ranged from 0.55 
to 2.20 with a mean value of 1.21; the h values ranged from -2. 12 to 0.45 with a mean value of 
-0.84. 

3. In order to determine the ideal item-response patterns corresponding to the attribute 
mastery patterns, the BUGLIB program (Varadi & Tatsuoka, 1989) was used. As a result, 461 
ideal item-response groups (representing 461 different knowledge states) were generated for 
method I, and 453 for method II. 

Results 

A. Method I classiFicatjon results 

The classification of the actual students* response patterns into the 461 predetermined 
knowledge states resulted in 55 non-empty groups. A summary of the classification results is 
presented in Table 3. As can be seen in the table, 15 groups had frequencies of 2 or more, the 
maximum having 10 students in a group. The table also presents the states into which one or more 
students were classified, ordered by IRT 0. Figure 1 is a tree representation of those states. Each 
state is represented by a node indicating the non-mastered attributes in that knowledge state, and 
located on the IRT 9-value scale, which is given on the left side of the table. The arcs connecting 
the nodes indicate transitional relationships among the states. A transition from one state of 
knowledge to another is said to be possible whenever the set of non-mastered attributes associated 
with the second state is a proper subset of the first state. Thus, arcs connect lower knowledge 
states to higher ones, where a higher state is defined as a state having at least one less non- 
mastered attribute than the lower state connected to it. 
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Insert Table 3 and Figure 1 about here 



B. Mftthod n cla ssification results 

The classification of the students' response patterns into the 453 predetermined knowledge 
states resulted in 51 non-empty groups. A summary of the classification results is presented in 
Table 4. As can be seen in the table, 20 groups had firequencies of 2 or more, the maximum having 
8 students in a group. The groups are ordered by IRT 0. Figure 2 is a tree representation of those 

states. 

Insert Table 4 and Figure 2 about here 

C. Classifying Students into the two Soliirion Methods. 

A decision rule was set to determine which of the two methods a given student was more 
likely to have used The shorter of the two distances (Mahalanobis' distances) between a student's 
response pattern and that of the nearest ideal item-response group in each method was chosen to 
indicate the student's group affiliation. Applying this decision rule resulted in 104 students being 
classified into method I and 89 into method H. Of the rest, 13 students had identical Mahalanobis' 
distances for both methods; 19 answered all items correctly, and 6 answered all items incorrectly ~ 
thus the method used by these students could not be determined. The students' average 
ability^roficiency levels as measured by IRT 6 were -0.08 and 0.05 for methods I and n, 
respectively (with SDs 0.98 and 0.84, respectively). Thus, the difference between the two groups 
in mathematics ability as inferred form their performance on the current test was insignificant. 
Among the students who were classified into methods I and 11, 80% and 81%, respectively, were 
within the 95% probability ellipses of knowledge states~the ideal response patterns (x^cv 3df a.05 
= 0.35). (For a complete discussion of probability ellipses in this regard see Tatsuoka & 
Tatsuoka, 1987). 
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Examples of Classified Responses for Method L 

To illustrate the outcomes of the rule space model for method I, three students who were 
better classified to this method are now described 

Student 13 correctly answered 6 items (items 6, 9, 14, 17, 23, 29) and erred on 26 items. This 
student was classified into knowledge state No. 437 with a Mahalanobis' distance of 0.00, 
indicating a perfect match between the student's response pattern and the ideal response pattern 
represented by that knowledge state. As can be seen in Table 3 the IRT 6 value for that state is a 
low -1.73, and it is characterized by non-mastery of the following attributes (sec Table 1): 1 
(adding a term to both sides of the equation), 2 (subtracting a term from both sides of the 
equation), 3 (applying order of operations), 4 (applying the distributive law), 5 (applying the 
commutative law), 7 (applying signed numbers opcratioiis), 8 (dividing across by the coefficient of 
X, resulting in x = b/a, when a =b), 10 (dividing across by the coefficient of x, resulting in x = b/a, 
when a >b), 12 (evaluating the equation), 13 (applying order of operations and the distributive 
law), and 14 (applying the symmetry law and evaluating the equation). 

In order to reach state No. 0 (mastery of all attributes) from the state the student is cunendy 
in (state 437), a number of transitions need to take place, as can be seen in the tree diagram 
presented in Figure 1 one possible path is through states 429 (in which students have mastered 
attributes 1, 2, 4, 12, and 14), to state 244 (attributes 3 and 5), to state 3 (attributes 8 and 10), to 
state 1, by which time one attribute remains to be mastered (7), thus reaching a mastery of all 
required skills (state 0). 

Student 50 correctiy answered 31 items and erred on item 21. This student was classified 
into knowledge state No. 1 with a Mahalanobis* distance of O.CX). As can be seen in Table 3, five 
other students were classified into this knowledge state, which has a 6 value of 1.23 and is 

characterized by non-mastery of only attribute 7 [Performing signed number operations]. 

Student 175 correctiy answered 25 items and erred on items: 1, 13, 16, 19, 21, 25, 30. 
This student was classified into knowledge state No. 301 with a Mahalanobis' distance of 0.04 
(the student's point is within the 99% probability ellipse for that state). As can be seen in Table 3, 
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six other students were classified into this state which is characterized by non*mastery of the 
attributes 1 [adding a term], 3 [order of operations], 7 [signed number operations], and 13 [order 
of operations and distributive law]* In order to reach state No. 0 (mastery of all attributes) fiom the 
state the student is currently in, a number of transitions need to take place, as can be seen in the tree 
diagram presented in Figure L One possible route is through states 3, 1 to state 0. 
Examples of Qassified Responses for Method 11 

To illustrate the outcomes of the rule space model for method H, three students who were 
better classified to this method using the above decisicm rule arc now described. 

Student 148 corrccdy answered 27 items and erred on 5 items (items 1, 13, 16, 25, and 30). 
This student was classified into knowledge state No. 234 with a Mahalanobis' distance of 0.(X), 
indicating a perfect match between the student's response pattern and the ideal response pattern 
represented by that knowledge state. As can be seen in Table 4 the IRT 6 value for that state is .51 
and it is characterized by non-mastery of attribute 3 (see Table 2): (applying order of operations). 
As can be seen in Appendix B, 61% of the subjects in method n group mastered that attribute. 

Student 136 correcdy answered 26 items and erred on the following 6 items [items 3, 5, 
26, 27, 28, and 31]. This student was classified into knowledge state No. 59 witii a Mahalanobis' 
distance of 0.05 (i.e., die student's point is within the 99% probability ellipse for that state). As 
caii be seen in Table 4, four other students were classified into this knowledge state, which has an 
IRT 9 value of -0.01 and is characterized by non-mastery of attribute 1 1 (dividing across by the 
coefficient of x, resulting in x = b/a, when a >b). As can be se>en in Appendix B, 57% of the 
subjects in method II group mastered that attribute. 

Student 142 correctiy answered 1 1 items (items: 2, 6, 8, 14, 17, 18, 20, 22, 23. 24. 29) 
and erred on the other items. This student was classified into knowledge state No. 83 with a 
Mahalanobis' distance of 0.09 (a value witiiin the 99% probability ellipse for that state). This state 
is characteri7>5d by non-mastery of the attributes 1 (adding a term to both sides of the equation), 4 
(applying the distributive law], 6 (applying the distributive aiid commutative laws), 8 (applying 
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signed number operations), and 1 1 (dividing across by die coefficient of x, resulting in x = b/a, 
when a >b). No other student in our sample was classified into that state. 
D. Comparing the Results of the two Solutior. Methods . 

The two methods, I and n, yielded overall significandy different results for item difficulties 
as was indicated by a discriminant analysis. Thirty five percent of the variance in item difficulty 
was explained by group affiliation to method I or n (Wilks Lamda 0.65, 32df = 76.01, 
p<.0001). The discriminant function yielded substantive weights (>3) for the following items: 8 
(.62), 16 (.57), 18 (.35), 21 (.55), 27 (-.33), 28 (-.34) , 29 (-.30). (The values in die parentheses 
are the standardized canonical discriminant function coefficients). As is evident from the signs of 
these weights, some items turned out to be easier for method I students and others for method n 
students. Item difficulties (percent conect) for each method appear in Appendices A and B. 
The mastery level for the two groups also differ as can be seen by comparing the mastery level of 
similar attributes in the two groups given in Appendices A and B. These differences can not be 
tested statistically because even for the same attribute definition different items may apply in the 
two methods. However, a qualitative comparison of the interpretations based on mastery profiles 
for each method indicates that for students in method I the least mastered attributes (see table 1) are 
7 (Pferforming signed numbers), 13 (Applying both arithmetic order and the distributive law in the 
same equation), 10 (Dividing across by the coefficient of x, [x=b/a when a>b]) and 5 (Applying 
the commutative law); whereas for method n students (see Table 2), the least mastered attributes 
arc: 6 (Applying the distributive and commutative law), 4 (Applying the distributive law), 1 
(Adding a term to both sides of the equation)* and 1 1 (Dividing across by the coefficient of x, 
when a>b). 

Discussion 

This paper illustrated the use of rule space to diagnose student*s individual and group-level 
mastery of attributes related to linear algebra. Two different prc-specificd solution models were 
identified and students were classified according to them. One model was chosen to be more 
mathematically sophisticated and involves mastery of the symmetry law and the application of a 
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heuristic that allows for strategic decision making when solving the equation (i-e,, to delay writing 
the equation in standard ftom until the final step). The other model represents a solution that 
progresses in a more standard fashion in which all the x-terms are brought to the left-hand side of 
the equation, and the constants to the right Many other solution models could exist, of course. In 
order to test these models, unique Q matrices would have to be written. 

Of this sample of students, 104 were more likely to be using the heuristic approach, and 89 
students the standard approach. Supporting evidence for this distinction was found in that item 
difficulties differed for each Q matrix, indicating that the difficulty of an item is a function of the 
strategy used to solve it (since different attributes are called upon for each method). For example, 
attribute 7 (Performing signed numbers, negative subtraction and multiplication operations) posed 
the greatest difficulty for students classified as using the heuristic approach. This finding seems 
reasonable in that students who evaluate the equation to see whether it is easier (i.e., results in 
positive integer values) to bring x-terms to the right-hand side rather than to the left-hand side of 
the equation would generally not encounter operations involving negative numbers. Note that 
attribute 7 poses difficulty across all levels of ability (see Table 3). Attribute 13 also poses 
consistent difficulty (Applying both arithmetic order and the distributive law in the same equation). 
On the other hand, attribute 2 (Subtracting a term from both sides of the equation) causes difficulty 
for only the lowest ability students. For students using the standard approach, on the other hand, 
attribute 6 (Applying the distributive an commutative laws in the same equation) proved the most 
difficult 

When we consider the partial-mastery chart for students using the heuristic method (Figure 
1), we see how the transitional states are interrelated when they are linked as proper subsets one of 
the other. One approach to remediation using this chart is to first identify the knowledge state that 
best describes the target student Then, to consider the transition path that causes the least change 
as reflected on the ability measure, 0. Thus, a student classified to state 437 is more likely to 
respond to remediation that results in attaining state 429 (i.e., remediating attributes 1, 2, 4, 12, 
and 14), rather than to remediation that results in attaining state 303 (i.e., remediating attributes 2, 
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5, 8, 10, 12, and 14) since the latter state is associated with higher-ability students. For a more 
complete description of how to use transitional states for remediation purposes, see Tatsuoka and 
Tatsuoka (1992). 

At the whole-class level, a teacher using the current analyses would know that a significant 
number of students were most likely not using the heuristic method Therefore, the teacher could 
explicitly teach the evaluation heuristic, which wo^ild provide the students a choice of solution 
models, and would make algebra seem less mechanical and more mathematical. Concerning the 
class's performance on each attribute, the teacher could address each of the unmastered attributes 
using whole class instruction. Similar options would exist at the individual student level, in which 
the teacher could focus on the strategy-level decisions that the student is making or on remediation 
of the nonmastered attributes. 

Comparing the rule space and bugg v a pproaches . In recent years, cognitive scientists and 
psychometricians have contributed to the effort to better understand mathematics performance 
beyond simple indices (e.g., Birenbaum & Tatsuoka, 1987; Brown & Burton, 1978; Matz, 1982; 
Sleeman, 1984; Tatsuoka, 1990; Tatsuoka & Tatsuoka, 1992; VanLehn, 1990). An alternative 
modeling approach to rule space is the buggy approach, in which diagnoses arc generated in 
response to the student's errors (Sleeman, Kelly, Martinak, Ward & Moore, 1989; Payne & 
Squibb, 1990; VanLehn, 1982). Many such errors may be "wild" or result from slips (e.g., 
Sleeman et al., 1989). As a consequence, remediation resulting from buggy analysis may lead the 
teacher and student far afield from the target task. To illustrate, consider an equation in the form ax 
= b. Bugs that have been noted for this case generate x = b (Sleeman et al., 1989), x = b - a 
(Sleeman et al., 1989; Payne & Squibb, 1990), x = -(a + b) (Gutvirtz, 1989), x = a - b (Gutvirtz, 
1989), and X = a + b (Gutvirtz, 1989; Payne & Squibb, 1990). To explain each of these cases, the 
teacher must make complex inferences about the underlying mathematical models of the student, 
and design remediation targeted to these inferences - predicated on the questionable assumption 
that students are not generating many of these errors capriciously (Sleeman et al., 1989; Payne & 
Squibb, 1990). 

to 
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The rule space analysis, by contrast, focuses diagnosis and remediation decisions on 
attributes that arc integral to the task at hand Then rule space analysis considers the extent to 
which the attributes for a given item are mastered over the entire test For item 7 in the test [8 + 
4(x - 3) = 24] method II, for example, the attributes to consider for this item would be 1 (Adding a 
term to both sides of the equaticHi), 4 (Applying the distributive law), 5 (Applying the commutative 
law), 6 (Applying the distributive and commutative law), 8 (Performing signed numbers 
operations), and 10 (Dividing across by the coefficient of x, when a<b [x=b/a]). The decision as 
to which attributes would be remediated would be based not on the given student's bug(s) for that 
item, rather on an analysis of how the attributes were mastered across the entire set of items by that 
student In addition, the information gathered on the entire sample of students allows the teacher to 
consider a pathway to mastery for this student by considering the number of students assigned to 
each knowledge state (see Table 4 and Figure 2). The usefulness of remediation based on these 
knowledge states remains to be tested empirically. If they are found to be of value instructionally, 
remedial strategies can be proposed and scripted beforehand to address nonmastery of each of the 
attributes. Further, the rule space analysis permits the investigation of the application of these 
attributes at a strategic level (heuristic vs. standard methods in this case), which lends itself to 
remediation at this level . Finally, a careful examination of the Q matrix and the resulting group 
attribute mastery profiles can aid in designing future tests in that topic, thus increasing the validity 
of those tests. Regarding questions of validity, it should be noted that the two Q matrices 
(describing two different approaches to solving the linear equations) resulted in different item 
difficulties. 
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Tabk Attributes Used to Describe Method L 
No. Description 

1 Adding a term to both sides of the equation 

2 Subtracting a term from both sides of the equation 

3 Applying arithmetic order of operations 

4 Applying the distributive law 

5 Applying the ccxnmutative law 

6 Adding or subtracting variable terms 

7 Performing signed numbers, negative subtraction and multiplication operations 

8 Dividing aciuss by the coefficient of x, [resulting in x=b/a when a=b] 

9 Dividing across by the coefficient of x, [resulting in x=b/a when a<b] 

10 Dividing across by the coefficient of x, [resulting in x=b/a when a>b] 

1 1 Applying symmetry law 

12 Evaluating the equation to determine the simplest solution path 

13 Applying both arithmetic order and the distributive law in the same equation 

14 Applying symmetry law and evaluating the equation to determine the simplest solution path 
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I^bk^. Attributes Used to Describe Method 
No. Description 

1 Adding a tenn to both sides of the equation 

2 Subtracting a term from both sides of the equation 

3 Applying arithmetic (xdcr of operations 

4 Applying the distributive law 

5 Applying the oxnmutative law 

6 Applying the distributive and conmiutative law 

7 Adding or subtracting variable terms 

8 Pferfonning signed numbers, negative subtraction and multiplication operations 

9 Dividing across by the coefficient of x, [resulting in x=b/a when a=b] 

10 Dividing across by the coefficient of x, [resulting in x=b/a when a<b] 

1 1 Dividing across by the coefficient of x, [resulting in x==b/a when a>b] 

1 2 Number of distinct mathematical operations > 3 

13 Multiplying both sides of the equation by (-1) 
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Table 3 . For Method I the States with two or More Students Classified into Them Ordered by 
Theta (0), and a Listing of Attributes Not Mastered 



State No. 


iRTe 


Frequency 


Attributes not Mastered 


0* 


5.00 


19 


(all mastered) 


1 


1.23 


6 


7 


3 


.41 


6 


7. 13 


6 


.12 


3 


4, 7, 13 


11 


.02 


3 


4, 5, 7, 13 


86 


-.21 


3 


7, 10 


107 


-.52 


2 


1. 5, 7, 10 


180 


-.13 


2 


7, 8, 14 


244 


-.57 


10 


7, 8, 10, 13 


301 


.13 


7 


1. 3, 7, 13 


303 


-.08 


3 


1. 3, 4, 7, 13 


304 


.12 


2 


3, 5, 7, 13 


348 


-.59 


7 


3, 5, 7, 10, 13 


376 


-.24 


2 


3, 4, 5, 7, 8, 13 


429 


-.73 


5 


3, 5, 7, 8, 10, 13 


437 


-1.73 


3 


1, 2, 3, 4, 5, 7, 8, 10. 12, 13, 14 



372* -5.00 6 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14 (none mastered) 



* Students in these states were not included in the analysis since their method could not be 
detennined 
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JahlsA' For Method H: The States with two or more Students Qassified into them Ordered by 
Theta (6), and a Listing of Attributes Not Mastered 



State No. 


6 


Frequency 


Attributes not Mastered 


0* 


5.00 


19 


(aU mastered) 


1 


.34 


2 


12 


3 


.33 


8 


1.6 


10 


.40 


5 


4 


11 


.27 


2 


4. 6 


12 


.00 


2 


1. 4,6 


14 


-.12 


2 


1. 4, 6, 12 


59 


-.01 


5 


11 


61 


-.15 


2 


6, 11 


73 


-.66 


2 


1, 4, 6, 11, 12 


213 


-1.79 


2 


1,4, 5, 6. 7, 8, 9, 11, 12 


234 


.51 


2 


3 


237 


.10 


2 


1. 3, 6 


244 


.09 


5 


3,4 


246 


-.11 


4 


1, 3, 4, 6 


285 


-.48 


2 


1. 3, 6, 11 


293 


-.56 


2 


3, 4, 6, 11 


294 


-.65 


2 


1. 3,4, 6, 11 


304 


-1.01 


2 


1,3, 4, 6, 8, 11 


336 


-.15 


2 


3, 5, 6, 9 


394 


-.57 


3 


3, 9, 11, 12 


453* 


-5.00 


6 


1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13 (none mastered) 


* Students in these states were not included in the analysis since their method could not be 
determined. 
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Appendix A 

The Incidence Matrix for Method I for the 32 Items Using 14 Attributes with Percent Correct 
for each item and Percent Mastered for each Attribute. 



Item 


Attributes 


% Correct 


11111 

12345678901 234 


Method I 


TTotfll ^amnlft 


1 


3+x«6+3*2 


01 10 00 00000000 


72 


74 


2 


7x+7«U 


01000001000000 


83 


81 


3 


16xM 


00000000010000 


57 


63 


4 


6x«2x-i-3 


0100010 00 10100 


58 


63 


5 


4x+21"=10x+17 


01000100011101 


52 


60 


6 


35:c7x 


00000000101000 


95 


93 


7 


844(x-3)=24 


11010000100000 


67 


73 


8 


3+6x=18 


0 1000000 100000 


69 


77 


9 


60+12=6x+2x 


00000100101000 


88 


81 


10 


4(2x+3)=10x 


01010100101101 

wX\#X\#X\#\#X\#X XwX 


84 


83 


11 


644x+x=22 


01000100100000 


75 


77 


12 


98=7+7x 


01000000101000 


85 


83 


13 


x-4=4+2*4 


10100000000000 


71 


73 


14 


llx-3x+4x=44.1244 


00000100 100000 


87 


87 


15 


4x+2=5+3x 


01000100000100 


84 


84 


16 


2+2*3(2x+3)=22x 


01110100101111 


27 


48 


17 


6x+8x^8^8 


00000100100000 


79 


81 


18 


8+4x=26 


01000000100000 






19 


6(x+3)=12x 


01010100101101 


80 


81 


20 


5+3x+x=16 


01000100100000 


75 


76 


21 


3+2*2(2x.3)=23x 


11111110011111 


20 


42 


22 


75=5+5x 


01000000101000 


85 


84 


23 


24>:6x 


00000000101000 


95 


92 


24 


12x+12=24 


01000001000000 


83 


81 


25 


4+x=6+2*3 


01 100000000000 


74 


73 


26 


8x=4X+2 


01000100010100 


66 


68 


27 


28x=7 


00000000010000 


53 


54 


28 


!4x+30=78-2x 


1 1000100100000 


80 


78 


29 


5x+2x-3x=25+12-9 


00000100100000 


91 


88 


30 


x.6=3+5*3 


10100000000000 


65 


67 


31 


7+4x=28x 


0100010001 1101 


47 


53 


32 


6-Hi(x.2)=18 


1 1010000100000 


66 


70 



%Mastered 69665905959827 

44498519615937 



2: 
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Appendix B 

The Incidence Matrix for Method n for the 32 Items Using 13 Attributes with Percent Conect for 
each item and Percent Mastered for each Attribute. 



Item 




Attributes 


% Correct 






nil 

1234567890123 


Method n 


Total sample 






0110000000000 

\J X xvwwvwvvvvv 


7^ 


lA 




7x+7=14 


0100000010000 


fi A 

oU 


fit 
oX 


3 


16x=:4 


0000000000100 




OD 


A 
*♦ 




000000 1000100 


DZ 


£.1 
OD 


5 


4x+21=10x+17 

A 1 X ^ X \/A~ X # 


0100001100101 


di 1 

ol 


£f\ 

ou 


5 


35=7x 


0100000001001 


y / 




7 




1001110101000 


/o 


15 


g 


3+6x=18 


0100000001000 




/ / 


Q 


WX 1 V AT^A 


0100001 101001 


0 1 
O 1 


fit 
ol 


1 n 

1 V 


HyXAT J 1 \/A 


0101001 101001 

VXWXwVX xwxwvx 


O A 


o3 


y 1 

A 1 


V T A^A^ArA 


0100001001000 


TO 


/ / 


1 0 


QR=7-i-7t 


0100000101001 

U X V/ w V/ w V/ X w X w V/ X 


o2 


o3 


1 J 


A ' 


1010000000010 


11 

15 


11 
15 


1 d 


X 1 A~<^ AX^A— 1 


0000001001010 


on 


ft7 
o / 




H AT^— J T J A 


OlOOOOlOOOOOO 


Q < 


54 






0111001101011 


o2 


^fi 


17 


6x-i-8x=48448 


0000001 00 1 000 


82 


81 


18 


8-f4x=26 


010000000 1000 


89 


85 


19 


6(x+3)=12x 


0101001 101001 


83 


81 


20 


5+3x+x=16 


0100001001000 


76 


76 


21 


3+2*2(2x-3)=23x 


1101111101011 


52 


42 


22 


75=5+5x 


0100000101001 


85 


84 


23 


24:«6x 


0100000001001 


92 


92 


24 


12x+12=24 


010000001 0000 


79 


81 


25 


4+x=6+2*3 


01 10000000000 


70 


73 


26 


8x=4X+2 


0100001000100 


65 


68 


27 


28x=7 


0000000000100 


44 


54 


28 


14x+30=78-2x 


1100001001000 


73 


78 


29 


5x+2x-3x=25+12-9 


0000001001000 


87 


88 


30 


x-6=3+5*3 


1010000000010 


66 


67 


31 


7+4x=28x 


0100001100101 


52 


53 


32 


6+4(x-2)=18 


0001110001000 


71 


70 



% Mastered i i 

5064729870579 
1012681590728 
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Hguie Captions 

Figure 1 

A Tree Representation of the States in Method I to Which More Than One Student Was Classified 
Note: The small numerals correspond to the State labels. 
Figure 2 

A Tree Representation of the States in Method n to Which More Than One Student Was Classified 
Note: The small numerals correspond to the State labels. 



Authors' note: 

The authors would like to thank Yaffa Gutvirtz for use of her data set for this study. 
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